Skip to content

fix(python): use non-deprecated params in search benchmark fixtures - #3

Closed
sezruby wants to merge 1 commit into
mainfrom
fix-bench-legacy-format
Closed

sezruby wants to merge 1 commit into
mainfrom
fix-bench-legacy-format

Conversation

@sezruby

@sezruby sezruby commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Fork-internal review PR. Targets sezruby:main for review before opening against lance-format/lance.

Problem

The dataset fixtures in python/benchmarks/test_search.py pass deprecated parameters to the dataset APIs:

  • lance.write_dataset(..., use_legacy_format=False)
  • lance.dataset(..., index_cache_size=64 * 1024)

The test config sets filterwarnings = ['error::DeprecationWarning', ...], so these emit DeprecationWarning as errors during fixture setup. As a result every benchmark in test_search.py errors out before running:

DeprecationWarning: use_legacy_format is deprecated, use data_storage_version instead
DeprecationWarning: The 'index_cache_size' parameter is deprecated. Use 'index_cache_size_bytes' instead.

Fix

Switch to the current parameters:

  • use_legacy_format=False → data_storage_version="stable" (the exact mapping the deprecation shim applies).
  • index_cache_size=64 * 1024 → index_cache_size_bytes=512 * 1024 * 1024 (512 MiB comfortably caches these 100k-row IVF_PQ indices).

Verification

$ uv run --group benchmarks pytest python/benchmarks/test_search.py::test_ann_no_refine --benchmark-only
test_ann_no_refine[clean]               541.48 us   1813 ops
test_ann_no_refine[with_delete_files]   770.46 us   1285 ops
test_ann_no_refine[with_new_rows]      2843.17 us    349 ops
3 passed

ruff check / ruff format --check clean.

The search benchmark dataset fixtures pass `use_legacy_format=False` and
`index_cache_size=...` to `write_dataset` / `lance.dataset`, both of which are
deprecated. Because the test config sets `filterwarnings = error::DeprecationWarning`,
fixture setup raised `DeprecationWarning` and every benchmark in
`test_search.py` errored out before running.

Switch to the current parameters (`data_storage_version="stable"` and
`index_cache_size_bytes`), restoring the benchmark suite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added A-python bug Something isn't working labels Jun 15, 2026
Xuanwo added a commit that referenced this pull request Oct 28, 2026
…rmat#8878)

## What this changes

A `linalg-asan` job in `nightly_run.yml` runs `cargo test -p
lance-linalg --tests` under `-Zsanitizer=address`.

The public entry points already validate the lengths a caller passes:
`sum_4bit_dist_table` asserts `dists.len() >= n`, and
`hamming_batch_u64` asserts the output length with a comment explaining
why it is not a `debug_assert`. What nothing checks is the pointer
arithmetic the kernels then do inside those bounds.
`_mm256_storeu_si256(dists.as_mut_ptr().add(16), ...)` in the AVX2
dist-table kernels and `*results_ptr.add(1)` through `.add(3)` in
`hamming_batch_avx2` are raw stores at offsets the kernel computes
itself, so an off-by-one in a chunk count writes past the buffer while
every entry assert still passes.

It lands in `nightly_run.yml` rather than `rust.yml` because
`-Zsanitizer` needs a nightly toolchain, and the only nightly this repo
uses is in `daily-code-coverage.yml`, which is `schedule` plus
`workflow_dispatch`. The pin is the same one that file uses, so there is
one nightly date in the repo rather than two.

Three details in the job are load-bearing and each is there for a reason
a reader would otherwise have to rediscover.

`RUSTUP_TOOLCHAIN` rather than `rustup default`. A rustup override file
outranks the default toolchain, and `rust-toolchain.toml` pins a stable
release, so `rustup default nightly-...` followed by `cargo test` runs
the sanitizer flag into a stable compiler and stops at `error: the
option 'Z' is only accepted on the nightly compiler`.
`daily-code-coverage.yml` solves the same problem the other way, with
`cargo +nightly-...`.

`RUSTFLAGS` repeats the baseline from `.cargo/config.toml`. Setting the
variable replaces that file's per-target `rustflags` wholesale rather
than appending, so leaving it at just the sanitizer flag would
instrument a build no release ships, and would compile out the arms
gated on `target_feature = "avx2"`. One of those is where the batch call
that writes into a heap buffer lives (`cosine_batch_avx512` into
`vec![0.0; batch.len() / dimension]`), which is the case a sanitizer
detects best.

`--target x86_64-unknown-linux-gnu` keeps the sanitizer off build
scripts and proc macros, which run on the host.

There is no protoc step, unlike every other Rust job here.
`lance-linalg`'s dependency closure contains no `prost-build` or
`tonic-build`, dev-dependencies included, and the job runs green without
it.

## Test plan

On aarch64-apple-darwin, with one deliberate 2-byte store planted past
the 32-slot output of `sum_dist_table_32bytes_batch_neon`, ASan reports
`heap-buffer-overflow, WRITE of size 2` at `dist_table.rs` with the
chain up through `sum_4bit_dist_table_uninit` to the test. The same
binary without the sanitizer dies with a bare `SIGTRAP` and no location,
so what this job adds on that mutation is the location rather than the
detection.

It has run for real on Linux, four times, on a fork branch with the
`github.repository` guard dropped since that guard is what stops it
running anywhere but here.

Two clean runs:
[33305364459](https://github.com/LuciferYang/lance/actions/runs/33305364459)
and, after dropping the protoc step,
[33312991796](https://github.com/LuciferYang/lance/actions/runs/33312991796).
Both green, about a minute and a half to build and three to four minutes
to run.

Then two runs with a deliberate out-of-bounds store planted in a kernel,
to check the claim that this job names a file and a line. That is where
the `ASAN_SYMBOLIZER_PATH` step comes from. Without it the report was
detected but useless:

```
WRITE of size 4 at 0x7cb5cf675100 thread T428
    #0 0x5620f81ba20f  (/home/runner/work/lance/lance/target/.../lance_linalg-...+0x...)
    #1 0x5620f81b1619  (/home/runner/work/lance/lance/target/.../lance_linalg-...+0x...)
```

ubuntu-24.04 ships `llvm-symbolizer` only under a versioned name, and
ASan looks for the unversioned one on `PATH`. With the step, the same
overflow reports:

```
WRITE of size 4 at 0x7ce193dcf280 thread T428
    #0 ... sum_hacc_dist_table_32bytes_batch_avx2 rust/lance-linalg/src/simd/dist_table.rs:479:5
    #1 ... sum_4bit_hacc_dist_table_avx2          rust/lance-linalg/src/simd/dist_table.rs:361:21
    #2 ... sum_4bit_hacc_dist_table_uninit        rust/lance-linalg/src/simd/dist_table.rs:244:13
    #3 ... sum_4bit_hacc_dist_table               rust/lance-linalg/src/simd/dist_table.rs:214:14
    #4 ... tests::test_sum_4bit_hacc_dist_table_matches_u16_reference_multi_batch
```

So the failing test names itself in the stack, which is why the run does
not need `--test-threads=1` to attribute a report.

That exercise also turned up a coverage limit worth stating. I first
planted the store in `sum_dist_table_32bytes_batch_avx2` and the job
stayed green: on a runner with AVX-512BW, `sum_4bit_dist_table_uninit`
selects the AVX-512 C kernel, so the Rust AVX2 one never executes. The
hacc entry point has no AVX-512 kernel, which is why the planted store
there did fire. So on an AVX-512 runner this job covers the hacc AVX2
kernel, the scalar paths and everything else in the crate, but not the
AVX2 dist_table kernel that a smaller host would take.

One gap remains: the C is not instrumented. `dist_table.c` compiles
without `-fsanitize=address`, so the AVX-512 dist_table kernel is not
covered. I tried `CFLAGS=-fsanitize=address` and it broke a host link in
`lance-arrow`'s build script, so it is left out rather than
half-applied.

## Scope note

`--tests` selects the lib's unit tests and every integration target, so
`tests/panic_location.rs` runs under the sanitizer too, and a target
added later is included without editing this job.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
@sezruby

sezruby commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Merged upstream in lance-format#7285.

@sezruby sezruby closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant